Skip to content

Move from custom DALI conda recipe to upstream nvidia-dali-feedstock - #6402

Open
JanuszL wants to merge 10 commits into
NVIDIA:mainfrom
JanuszL:move_to_feedstock
Open

Move from custom DALI conda recipe to upstream nvidia-dali-feedstock#6402
JanuszL wants to merge 10 commits into
NVIDIA:mainfrom
JanuszL:move_to_feedstock

Conversation

@JanuszL

@JanuszL JanuszL commented Jun 24, 2026

Copy link
Copy Markdown
Contributor
  • uses nvidia-dali-feedstock recipe to build DALI in CI
  • removes DALI maintained recipe

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

  • uses nvidia-dali-feedstock recipe to build DALI in CI
  • removes DALI maintained recipe

Additional information:

Affected modules and functionalities:

  • conda build process

Key points relevant for the review:

  • NA

Tests:

  • Existing tests apply
    • all test *_conda
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: DALI-4680

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55681719]: BUILD STARTED

@greptile-apps

greptile-apps Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR replaces DALI's custom conda recipe (conda-build + mambabuild) with the upstream nvidia-dali-python-feedstock and rattler-build. A patch file adapts the feedstock for local source builds, and build_conda_packages.sh is rewritten to clone the feedstock, apply the patch, swap the release-tarball source for a local path, then drive rattler-build.

  • Recipe removal: All files under conda/dali_native_libs/ and conda/dali_python_bindings/ (build scripts, meta.yaml, conda activate/deactivate hooks) are deleted in favour of the upstream feedstock.
  • New orchestration: build_conda_packages.sh adds input validation for CUDA_VERSION_MAJOR, DALI_VERSION, DALI_TIMESTAMP, and NVIDIA_BUILD_ID, applies nvidia-dali-python-feedstock.patch with --check before actual application, and verifies every sed substitution with grep -q guards before invoking rattler-build.
  • Patch file: Updates libnvimgcodec from 0.8.* to 0.9.* across all .ci_support/ YAML variants, adds BUILD_FOR_CONDA=ON/BUILD_TEST=ON/BUILD_BENCHMARK=ON, and injects build-flavour/timestamp context vars into recipe.yaml.

Confidence Score: 5/5

Safe to merge — this is a build-infrastructure-only change that removes the in-tree conda recipe and delegates to the upstream feedstock, with all previously raised issues now resolved.

All previous review findings (unpinned clone, silent sed, unguarded CUDA variable, wrong git -C path, missing dlpack/cute/cutlass verifications) are addressed. The new script validates every required input before use, applies the patch with --check first, and guards each sed substitution with a confirming grep. No logic regressions are introduced in the DALI core or Python bindings.

Files Needing Attention: No files require special attention. The two actively executed files — build_conda_packages.sh and nvidia-dali-python-feedstock.patch — are both well-guarded.

Important Files Changed

Filename Overview
conda/build_conda_packages.sh Rewritten to use rattler-build + upstream feedstock; adds explicit input validation, patch --check guard, and grep verifications for every sed substitution. All previously-flagged issues are addressed.
conda/nvidia-dali-python-feedstock.patch New patch file: removes release-tarball sources/sha256/patches, adds rm-rf cleanup before symlink creation, enables BUILD_FOR_CONDA/BUILD_TEST/BUILD_BENCHMARK, bumps libnvimgcodec 0.8 to 0.9 across all CI support YAMLs, and injects DALI version/flavor env vars into recipe.yaml context.
conda/config/conda_build_config.yaml Deleted; equivalent configuration now lives in the upstream feedstock's recipe/conda_build_config.yaml (patched by nvidia-dali-python-feedstock.patch).
conda/dali_native_libs/recipe/build.sh Deleted; CMake invocation logic moved to the upstream feedstock's build.sh (with DALI-specific flags injected via the patch file).
conda/dali_native_libs/recipe/meta.yaml Deleted; dependency pinnings and package metadata now managed by the upstream feedstock recipe.
conda/dali_python_bindings/recipe/build.sh Deleted; Python bindings build logic is handled by the upstream feedstock.
conda/dali_python_bindings/recipe/meta.yaml Deleted; Python bindings package metadata now managed by the upstream feedstock recipe.
conda/dali_native_libs/recipe/activate.d/nvimgcodec_extensions_path.sh Deleted; nvimgcodec extensions-path workaround activate hook removed along with the old recipe.
conda/dali_native_libs/recipe/deactivate.d/nvimgcodec_extensions_path.sh Deleted; counterpart deactivate hook for the nvimgcodec extensions-path workaround removed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[build_conda_packages.sh] --> B{Validate inputs}
    B -->|invalid| ERR1[exit 1]
    B -->|valid| C[Select METAL_YAML]
    C --> D[git clone feedstock]
    D --> E{DALI_FEEDSTOCK_SHA set?}
    E -->|yes| F[git checkout SHA]
    E -->|no| G[use HEAD]
    F --> H[git apply --check patch]
    G --> H
    H -->|fail| ERR2[exit 1]
    H -->|OK| I[git apply patch]
    I --> J[sed: URL to local path]
    J --> K{grep verify}
    K -->|fail| ERR3[exit 1]
    K -->|OK| L[sed: use_gitignore false]
    L --> M{grep verify}
    M -->|fail| ERR4[exit 1]
    M -->|OK| N[rattler-build build]
    N --> O[Copy artifacts]
Loading

Reviews (16): Last reviewed commit: "Update conda CI nvImageCodec pins" | Re-trigger Greptile

Comment thread conda/build_conda_packages.sh
Comment thread conda/build_conda_packages.sh Outdated
Comment thread conda/build_conda_packages.sh Outdated
@JanuszL
JanuszL force-pushed the move_to_feedstock branch from 957c7ca to 714dac3 Compare June 24, 2026 12:18
Comment thread conda/build_conda_packages.sh Outdated
@JanuszL
JanuszL force-pushed the move_to_feedstock branch from 714dac3 to dd7d6ca Compare June 24, 2026 12:24
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55681719]: BUILD FAILED

@JanuszL
JanuszL force-pushed the move_to_feedstock branch from dd7d6ca to 8fa4e57 Compare June 24, 2026 12:44
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55685089]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55685089]: BUILD FAILED

@JanuszL
JanuszL force-pushed the move_to_feedstock branch from 8fa4e57 to f63cb59 Compare June 24, 2026 13:43
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55688559]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55688559]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55693916]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55694466]: BUILD STARTED

Comment thread conda/build_conda_packages.sh Outdated
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [55696184]: BUILD STARTED

@JanuszL
JanuszL marked this pull request as draft June 24, 2026 16:00
@JanuszL
JanuszL force-pushed the move_to_feedstock branch 2 times, most recently from 3e8800a to a5acefb Compare July 13, 2026 16:55
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57832399]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57832399]: BUILD FAILED

@JanuszL
JanuszL force-pushed the move_to_feedstock branch from a5acefb to 86ccb4a Compare July 14, 2026 11:38
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57957810]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57957810]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57967556]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [57967556]: BUILD FAILED

@JanuszL
JanuszL force-pushed the move_to_feedstock branch from 86ccb4a to 536c992 Compare July 14, 2026 18:10
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58002631]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58002631]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58105433]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58105433]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58114829]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [58114829]: BUILD PASSED

@JanuszL
JanuszL marked this pull request as ready for review July 15, 2026 20:48
Comment thread conda/nvidia-dali-python-feedstock.patch
@JanuszL
JanuszL force-pushed the move_to_feedstock branch from f29a40a to 72e4bd3 Compare July 23, 2026 09:14
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59219619]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59219619]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59248913]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59248913]: BUILD FAILED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59272399]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [59272399]: BUILD PASSED

@stiepan stiepan self-assigned this Aug 3, 2026
JanuszL added 10 commits August 3, 2026 20:05
- uses nvidia-dali-feedstock recipe to build DALI in CI
- removes DALI maintained recipe

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
@JanuszL
JanuszL force-pushed the move_to_feedstock branch from dd2c65d to 2cc9de3 Compare August 3, 2026 18:05
@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [60894791]: BUILD STARTED

@dali-automaton

Copy link
Copy Markdown
Collaborator

CI MESSAGE: [60894791]: BUILD PASSED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants